Topological insulators (TIs) represent a new class of materials intensively studied recently.^[1,2] Up to date, most research in the field has been focused on band insulators with weak electron–electron interaction, such as typical V₂–VI₃ (e.g., Bi₂Te₃, Bi₂Se₃, Sb₂Te₃) series^[3,4] and III–V–VI₂ (TlBiTe₂ and TlBiSe₂) series^[5,6] compounds, in which the Dirac surface states stem as a result of the s–p band inversion.^[1,7] After these initial achievements, the search for TIs in correlated electron systems^[8] has attracted increasing research attention, as correlated TIs not only provide a new platform that bridges the topological non-trivial states and other exotic phenomena in correlated materials (such as the formation of the topological Mott insulator^[9,10] and topological crystalline insulator^[11]), but also provide a test ground to check the theoretical calculation, which has been mostly successful in predicting the weakly interacting TIs.^[1,2,7]

Recently, several rare-earth hexaboride compounds have been predicted to be correlated TIs or topological Kondo insulators (TKI), including SmB₆ and YbB₆.^[12,13] However, despite the intensive experimental effects on SmB₆ recently,^[14–18] the symbolic Dirac fermions formed by the surface state band (SSB) have not been well resolved in SmB₆ as in previously discovered TIs.^[1–8] In addition, although the transport measurements show evidence of dominating surface channels at low temperature, the fermiology extracted from the quantum oscillation of the dHvA effect^[19] does not match the Fermi pockets observed by angle-resolved photoemission spectroscopy (ARPES). The lack of SdH oscillation observation further makes the situation puzzling given the high quality of the SmB₆ samples.^[20] Moreover, there have been heavy debates about whether SmB₆ is a TKI or trivial insulator. While previous ARPES measurement suggested topological surface states with helical spin structure, a recent ARPES study provided a topologically trivial explanation for the observed band structure of SmB₆.^[21]

Under this circumstance, YbB₆ receives focused interest recently. Although similar to SmB₆, the electron correlation in YbB₆ is relatively weak,^[22] it can serve as an ideal model system to investigate the topologically non-trivial states in correlated electron system: on one hand, the interaction effect in YbB₆ is weaker than that in SmB₆, thus the electronic structure can be understood relatively easily; on the other hand, the interaction in YbB₆ is already strong enough thus the previous theoretical calculation^[12] has shown obvious deviation from the ARPES experimental observation. On the other hand, similar to the situation in SmB₆, there have been heavy debates regarding the topological nature of YbB₆.^[22–25] It is thus necessary to further study the electronic structure of this TKI candidate.

In this work, we studied the electronic structure of YbB₆ by both ARPES and torque magnetometry (TM) methods. We directly observed the SSB in YbB₆. Interestingly, the fermiology revealed by the quantum oscillation of TM measurements shows excellent agreement with the ARPES results. Moreover, the band structure we observed suggests that the band inversion in YbB₆ happens between the Yb_5d and B_2p bands — different from the previous calculation^[12] that suggested an inversion between the Yb_5d and Yb_4f bands. This difference clearly shows the effect of strong electron–electron interaction and the importance of experimental studies in correlated TI materials. With the input from our experiments, we were able to correct the previous theoretical calculation,^[12] paving a way to develop proper theoretical methods for exploring other strongly correlated TI materials.

The crystal structure of YbB₆ is shown in Fig. 1(a), where Yb forms a cubic lattice with a B₆-octahedron residing inside each unit. The Brillouin zone (BZ) is illustrated in Fig. 1(b) with high symmetry points labeled. The high quality of the samples is illustrated by the Laue characterization [Fig. 1(c)] and the powder x-ray diffraction [Fig. 1(d)]. Figure 1(e) illustrates the characteristic Yb_5p and Yb_4f core levels.

Fig. 1.

Figure Option ViewDownloadNew Window

Fig. 1. An overview of YbB6. (a) Crystal structure of YbB6. (b) The 3D and surface projected 2D Brillouin zones of YbB6, red and cyan dots indicate the high-symmetry points. (c) and (d) Laue pattern and powder x-ray diffraction showing the high quality of the crystals. (e) Core level photoemission spectrum showing the prominent 4fand 5p-electron characteristic peaks of Yb. The inset in (f) shows a flat sample surface along (001) direction. (f) Fermi surface of YbB6 in the first BZ obtained by integrating ARPES intensity in an energy window of 20 meV around EF. The blue lines indicate the projected surface BZ. (g) Zoom-in plots of FS maps measured using different photon energies at both and .

The Fermi surface (FS) map of YbB₆ in Fig. 1(f) shows two sets of distinct FS pockets at the and points, respectively: a large circular hollow pocket (FS area ∼ 0.04 Å⁻²) at and an elliptical hollow pocket (FS area ∼ 0.06 Å⁻²) centered at , each enclosing a smaller filled disk-like pocket inside. To check the nature of these FS pockets, we focus our measurements on each pocket and perform the photon-energy dependent ARPES measurements. In Fig. 1(g), the evolution of the FS pockets at both and is illustrated, which shows distinct behaviors of the outer hollow FS pockets and the inner disk-like pockets – while the outer hollow pockets centered at both and do not change their shape with photon energy, the inner disk-like pockets change dramatically with photon energy – indicating the 3-dimensional (3D) and 2-dimensional (2D) nature of the inner disk-like feature and out hollow pockets.

This difference in the dimensionality of the FS pockets can also be seen from the band dispersions. The band structure related to the FSs around is illustrated in Figs. 2(a)–2(c), where the 3D electronic structure and the band dispersion along the high symmetry direction are shown in Figs. 2(a) and 2(b), respectively. Around , there are sharp linear bands outside some fluffy bands, which can be clearly resolved in the momentum distribution curves (MDCs) in Fig. 2(c). This distinction between the dispersion sharpness comes from the different dimensionality of the bands: due to the k_z broadening,^[27] the 3D bands that have strong k_z-dispersion will smear up in the ARPES measurement, while the 2D bands do not suffer from such broadening and remain sharp. To further verify this, we show the broad range (40–90 eV) photon energy dependent measurements in Fig. 2(d). Indeed, the MDC peaks of the outer band form two straight lines, indicating no k_z-dispersion, or the 2D (surface) nature of the outside band. On the other hand, the inner pocket shows dramatic variation at different photon energies, confirming its bulk nature. The similar measurements near the points are shown in Figs. 2(e)–2(h) and the same conclusion can be drawn that the outer band originates from the surface and the inner one shows clearly the bulk character.

Fig. 2.

Figure Option ViewDownloadNew Window

Fig. 2. Electronic structure of YbB6 and identification of the surface states. (a) 3D illustration of the band structure of YbB6 around the point measured with photon energy hv = 65 eV. (b) Photoemission intensity plot along . The inset shows the direction of the cut. (c) Stacked plot of MDCs from the cut in panel (b). (d) Photon energy dependence of MDCs at EF along , confirming the weak kz dispersion of the band around . (e) 3D illustration of the band structure of YbB6 around the point measured with photon energy hv = 40 eV. (f) Photoemission intensity plots along (left) and along (right). The insets show the direction of the cuts. (g) Stacked plot of MDCs from the cut in the left panel of (f). (h) Photon energy dependence of MDCs at EF along , confirming the weak kz dispersion of the band around .

To examine the origin of the SSBs in YbB₆, we compare our observed band structure and the previous theoretical work,^[12] and discover a couple of clear discrepancies. In the previous ab initio calculation, the topological surface state originated from the band inversion between Yb_5d and Yb_4f as schematically illustrated in Fig. 3(a), thus the SSBs should emerge at the inversion band gap between the Yb_5d and Yb_4f bands and are very close to the flat 4f bands which were predicted to reside near the E_F.^[12] However, our measurements in a broad energy range do not show the Yb_5d band that hybridizes with the flat Yb_4f band. Actually, the SSBs near E_F are well separated from the Yb_4f band that resides about 1 eV below E_F [Figs. 3(c) and 3(d))]. These discrepancies suggest that the previous calculation may not have accounted for the electron correlation effect accurately and thus needs to be improved. It is interesting to note that the electronic structure and the origin of the surface state in SmB₆ are in strike contrast to those in YbB₆, although they share a common crystal structure. In SmB₆, the f orbital locates close to the Fermi level so that it can hybridize with the dispersive d orbital to induce a band inversion. The surface states emerge in the inverted band gap between the f and d orbitals, consistent with the mechanism shown in Fig. 3(a).

Fig. 3.

Figure Option ViewDownloadNew Window

Fig. 3. The origin of the surface states. (a) Schematic of the scenario proposed in Ref. [12], where the band inversion and inverted band gap take place between the Yb4f and Yb5d bands. (b) Schematic of the scenario where the band inversion and inverted band gap take place between the Yb5d and B2p bands. (c) Wide range photoemission intensity plot along . Curves on top show the ab initio calculation results with on-site Coulomb repulsion energy U = 8 eV. SSB: surface state band. (d) Zoom-in plots of ARPES spectra in the blue and orange rectangles in panel (c).

To reproduce the experiment results, we carry out further ab initio calculation, and find that with much increased on-site Coulomb repulsion energy U (up to 8 eV) compared to the previous study,^[12] we are able to reproduce the experimental results and get excellent match. The band inversion now occurs between the B_2p and Yb_5d bands [Fig. 3(b)], instead of the Yb_5d and Yb_4f bands [Fig. 3(a)] in the previous calculation.^[12] The much lowered Yb_4f bands now agree well with the experiments [Fig. 3(c)], and the overall band dispersions also match better with the measurements. We note that a similar result was reported in the same material, except that a much smaller on-site Coulomb repulsion energy U about 4 eV was used.^[24]

The experimentally observed bands and FSs are in good consistency with our ab initio calculation – the schematic of the calculated 3D FS is illustrated in Fig. 4(a) (top panel), which consists of two cylindrical surface state FSs (red color) centered at the and points and an elliptical FS pocket (yellow color) centered at each X point (at the face center of the BZ). In the middle panel, we show the projection of the 3D FS onto the k_x–k_y plane, which agrees excellently with the experimental results (bottom panel).

Fig. 4.

Figure Option ViewDownloadNew Window

Fig. 4. Magnetoresistance and dHvA oscillations in YbB6. (a) Illustration of the electronic structure of YbB6 in the 3D and 2D projected Brillouin zones. The bottom shows the FS obtained in our ARPES experiment. (b) The field dependence magnetic torque τ (upper panel) and its first derivative dτ/dH (lower panel) of YbB6. Oscillating patterns are observed in magnetic field higher than 8 T. The inset is the sketch of the experimental setup. The sample stage is rotated to tilt magnetic field H in the crystalline a–c plane. The magnetic field is applied to the sample with a tilt angle f relative to the crystalline c axis. (c) The oscillatory magnetic torques plotted as a function of 1/µ0H, clearly revealing the quantum oscillation pattern. (d) The main oscillation peak and its harmonic are observed in the FFT transformations of the derivative dτ/dH. The inset shows the corresponding FS pockets of each main oscillation peak, with the extracted area contour of the surface states at and labeled by the dotted green and blue curves, respectively. (e) The oscillatory torque signals measured at selected temperatures in magnetic field up to 35 T and tilt angle ϕ ∼ 10°. (f) The temperature dependence of the normalized FFT peak amplitudes for both pockets α and β. The effective mass is determined by fitting the temperature dependence to the thermal damping factor RT of the LK formula.

Given the general discrepancy between the ARPES results and the transport measurements in another correlated TI candidate SmB₆, we would like to check if the fermiology measured by quantum oscillation can be consistent with ARPES in YbB₆. We performed torque magnetometry measurements with the magnetic field close to the crystal (001) axis.^[26] The magnetic field tilt angle ϕ is about 0.5° [Fig. 4(b)]. The results are summarized in Fig. 4. The torque measurements reveal clear quantum oscillation in magnetization (the de Hass–van Alphen effect, or dHvA effect). Two major frequencies are observed: F_α = 400 T and F_β = 580 T [Figs. 4(b)–4(d)]. These oscillations arise from the Landau level quantization and the oscillation frequencies are related to the FS area A via the Onsager relation , where ϕ₀ = 2.07 · 10⁻¹⁵ T·m² is the flux quantum. Thus the corresponding FS volume is determined to be A_α = 3.8×10⁻² Å⁻² and A_β = 5.5×10⁻² Å⁻². These values are in good agreement with the surface state FS pockets around and observed in the ARPES [see the dashed circle and oval in the inset of Fig. 4(d)]. The slight FS area difference between the dHvA and ARPES may result from their different measurement conditions.

We note that figure 4(d) does not reveal the frequency corresponding to the smaller bulk FS (the inner pockets around and ), this may be due to the smaller mobility of the bulk electrons (as compared to the surface states). The much smaller Fourier transform (FFT) peak amplitude of the β pocket than that of the α pocket [Fig. 4(d)] may result from the much larger scattering between the surface and bulk electrons at the point. Indeed, as can be seen in the inset of Fig. 4(d), the inner and outer FSs around the point overlap much more than those around the point.

We further explored the temperature dependence of the dHvA oscillation amplitude. Figure 4(e) shows the oscillatory torque signal at temperature up to 10 K in magnetic field up to 35 T. The overall oscillation amplitude decreases with increasing temperature. It is described by the thermal damping factor in the Lifshitz–Kosovich (LK) formula , where the carrier effective mass is m^*m_e, m_e is the bare electron mass, and α = 2π²k_Bm_e/eℏ = 14.69 T/K. Figure 4(f) shows the normalized FFT amplitudes of the oscillating frequencies and the fitting based on the R_T factor of the LK formula. The effective mass of pocket α is about 0.15m_e and that of pocket β is about 0.27m_e. Based on the oscillation frequency and assuming the corresponding FS is of circular shape, we infer that the Fermi velocity of pocket α is , and that of pocket β is , also showing excellent agreement with the velocity data extracted from the ARPES experiment (5.6 eV·Å or 8.5×10⁵ m/s for the α pocket and 3.8 eV·Å or 5.76×10⁵ m/s for the β pocket along the axis) through linear fitting of the E_F vicinity electrons.

In summary, we have observed two sets of FSs by high resolution ARPES measurements in YbB₆ and acquired corresponding dHvA quantum oscillation for the surface part of the pocket. Our ARPES results help to improve the theoretical understanding regarding the mechanism of the gap opening of the inverted bulk bands and the origin of the surface states in YbB₆. Moreover, the discrepancy between the experiment and the previous theoretical calculation not only shows the vital importance of the experimental effect in the field, but also suggests that the theoretical calculation in correlated materials must be carried out with much caution, unlike in the weakly correlated TIs.